Skip to content

fix(remote): guard start against launching while the weights are seeding - #166

Open
outofcoffee wants to merge 3 commits into
mainfrom
seed-progress
Open

fix(remote): guard start against launching while the weights are seeding#166
outofcoffee wants to merge 3 commits into
mainfrom
seed-progress

Conversation

@outofcoffee

Copy link
Copy Markdown
Collaborator

spinloop remote start can no longer boot an instance against weights that are still being fetched: it checks the completeness record before any instance work, and while the weights are absent it reports the seed that is producing them.

Summary

  • The start Lambda checks the environment's weights against the seeding completeness manifest before launching or re-waking, and returns a retryable seeding state that names the seed while the weights are absent.
  • When no seed is in flight for those weights, the start starts one itself, using the same launch path as deploy, so the weights are produced rather than the start failing; a seed whose compute has ceased is re-run under the same identity on the next start.
  • The in-flight seed cap is enforced on the start path: at the cap the start reports the retryable state and waits for a slot instead of launching another seed.
  • Seed instance discovery now lives in remote/lambda/shared/seed/discovery.ts, shared by the seed and start Lambdas.
  • The Go start client prints seeding the weights (seed ) while it waits, and a deadline reached mid-seed gives up with an error that names the seed and the spinloop remote seed status command.
  • The fleet start phase stream gains a seeding phase rendered as seeding the weights (Xm Ys), so the dashboard and remote start count up while the fetch runs.
  • The start function's CDK environment and role gain the seed launch grants (seed env vars, PassRole on the seed role, the AL2023 AMI parameter, read on the weights prefix).

Implementation details

  • A stopped seed does not count as in flight, matching the seed command's convergence rules; only pending and running seeds hold the start and only those count against the cap.
  • A manifest read failure answers 502, as deploy does, because a control-plane error is not a wait.
  • deploy's own auto-seed does not enforce the cap; that predates this change and is left as a follow-up.
  • The openspec change guard-start-during-seed ships unarchived with this PR; its endpoint-lifecycle delta is synced and archived after it merges.
  • remote/vitest.config.ts: the stack-synth beforeAll hook now gets the same 120s headroom as the tests; it was flaking at the default 10s hook timeout under parallel load.

@outofcoffee outofcoffee added the bug Something isn't working label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant